[VL] Add a docker build job and reuse pre-built arrow libs #6826
Conversation
PHILO-HE
commented
Aug 14, 2024
•
PHILO-HE
commented
- Add a docker build job with the help of GHA. The DOCKERHUB_USER & DOCKERHUB_TOKEN secrets already set are used to login docker hub and push docker image. But these secrets are not available in pr from forked repo, which makes this pr's docker build job red.
- Use the new docker in CI checks. As arrow libs (static lib, jar) are installed in docker image, build_arrow option can be turned off.
|
Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues? https://github.com/apache/incubator-gluten/issues Then could you also rename commit message and pull request title in the following format? See also: |
PHILO-HE
force-pushed
the
build-docker-upstream
branch
from
7e523b0 to
8d0a771
Compare
PHILO-HE
force-pushed
the
build-docker-upstream
branch
from
8d0a771 to
9de4b10
Compare
| @@ -51,7 +51,7 @@ concurrency: | |||
| jobs: | |||
| build-native-lib-centos-7: | |||
| runs-on: ubuntu-20.04 | |||
| container: apache/gluten:gluten-vcpkg-builder_2024_08_05 # centos7 with dependencies installed | |||
| container: apache/gluten:vcpkg-centos-7 | |||
There was a problem hiding this comment.
Are we planning to do the same on centos 8 build?
There was a problem hiding this comment.
@zhztheplayer, yes, it would be better to also do this for centos-8 build. At least, those libs generated by arrow build (cpp/java) can be cached in docker to accelerate CI job when the current folder based cache miss happens.
There was a problem hiding this comment.
can we reuse the binary on centos 7?
There was a problem hiding this comment.
@zhouyuan, I will try this.
This reverts commit ffd33fb.
This reverts commit 3ba2feb.
|
===== Performance report for TPCH SF2000 with Velox backend, for reference only ====
|
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| name: Build and Push Docker Image |
There was a problem hiding this comment.
can you please help to add one doc on how to use this?
There was a problem hiding this comment.
@zhouyuan, I will do that. Thanks!
| @@ -2,12 +2,9 @@ | |||
|
|
|||
| set -e | |||
|
|
|||
| yum install sudo patch java-1.8.0-openjdk-devel -y | |||
There was a problem hiding this comment.
Initially this is kept in case someone is using this script from a clean centos 7 image
There was a problem hiding this comment.
@zhouyuan, maybe we can just tell user to refer weekly build job for building from clean image. centos-7/8/9, ubuntu-20.04/22.04 are covered in that job. I will document this guide. Thanks!
